feat(table): WKB encoding + GeoArrow conversion for geometry/geography - #1138
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
In general this looks good to me, though I'm not AS familiar with the geo stuff here.
@dwilson1988 or @paleolimbot would either of you be able to take a quick look here for verification?
paleolimbot
left a comment
There was a problem hiding this comment.
Cool!
The test cases from apache/arrow-rs#10065 are my most recent attempt at a succinct list of the special cases to take care of, although I've tried to note them inline here, too. The difference between Parquet and Iceberg for CRSes is that Iceberg requires authority:code (or PROJJSON that is offloaded into a table property). You can error for that case (I left inline suggestions about how to convert from PROJJSON to authority:code).
| func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata { | ||
| if crs == "OGC:CRS84" { | ||
| return geoarrow.NewMetadata() | ||
| } |
There was a problem hiding this comment.
This is reversed I think...if the iceberg CRS is "", the GeoArrow CRS is "OGC:CRS84" (unless NewMetadata().
| if strings.HasPrefix(strings.ToLower(crs), "srid:") { | ||
| id := crs[len("srid:"):] | ||
| raw, _ := json.Marshal(id) | ||
|
|
||
| return geoarrow.Metadata{ | ||
| CRS: raw, | ||
| CRSType: geoarrow.CRSTypeSRID, | ||
| } | ||
| } |
There was a problem hiding this comment.
There is one special case here: "srid:0" maps to an omitted GeoArrow CRS.
| raw, _ := json.Marshal(crs) | ||
|
|
||
| return geoarrow.Metadata{ | ||
| CRS: raw, | ||
| CRSType: geoarrow.CRSTypeAuthorityCode, | ||
| } |
There was a problem hiding this comment.
I believe there is one special case here: if the Iceberg CRS is projjson:<table property>, the CRS is the JSON object in that table property field. You should probably either support that or error with a note saying that it's not supported.
| if len(meta.CRS) == 0 && meta.CRSType == "" { | ||
| return "OGC:CRS84", nil | ||
| } |
There was a problem hiding this comment.
If the GeoArrow CRS is omitted from the extension metadata, the Parquet equivalent is "srid:0". If the GeoArrow CRS is "OGC:CRS84" or "EPSG:4326", the canonical Parquet CRS is "omitted" (i.e., default). I believe the Parquet and Iceberg CRS definitions are in sync now but it's worth double checking.
| switch meta.CRSType { | ||
| case geoarrow.CRSTypeSRID: | ||
| return "srid:" + crs, nil | ||
| case geoarrow.CRSTypeAuthorityCode: | ||
| return crs, nil | ||
| default: | ||
| return "", fmt.Errorf("unsupported geoarrow CRS type %q", meta.CRSType) | ||
| } |
There was a problem hiding this comment.
If the CRSType is PROJJSON, this is a important case to handle (because most GeoArrow producers produce that). You can convert most of these to authority:code by looking for the id member that looks like this "crs": {..., "id":{"authority": "OGC", "code": "CRS84"} or this "crs": {..., "id":{"authority": "EPSG", "code": 3857}. It would also be a good idea to convert the two "lonlat" CRSes ("EPSG:4326" and "OGC:CRS84") to the Iceberg "default" canonically.
When you can't extract an authority:code, this would need to be written to a table property and written to the CRS field as (projjson:<the table property name>). Probably easier to error for that case for now.
Happy to re-review if needed, but looks like @paleolimbot is already on it! Ping me if you want a second set of eyes. |
|
Thank you @paleolimbot and @zeroshade for reviewing. I believe I have addressed each comment by:
I would appreciate a second pass whenever you guys have a chance, thanks! |
laskoviymishka
left a comment
There was a problem hiding this comment.
Nice work threading the WKB / GeoArrow metadata through the schema conversion. The SRID encoding, the planar vs spherical edge handling, and the round-trip tests are all looking solid.
There are two things I’d want fixed before merge.
First, the EPSG:4326 handling looks asymmetric. geoArrowCRSToIcebergCRS collapses EPSG:4326 to OGC:CRS84 on read, but icebergCRSToGeoArrowMetadata writes EPSG:4326 back out as-is. Those two are not interchangeable because the axis order is different. That means a geometry("EPSG:4326") field — which is what PyIceberg emits — silently comes back as geometry("OGC:CRS84"). So the round trip is not actually equal. The existing CRS test uses EPSG:4267, so it misses this branch. I think we should either make the conversion symmetric or stop collapsing it, and add a regression test specifically for EPSG:4326.
Second, the projjson behavior is also asymmetric. The read path returns a clean error, but the write path panics. That panic propagates through VisitGeometry / VisitGeography with no recovery, so converting a table with a projjson CRS can crash the reader. This should return an error instead.
I left a few smaller comments inline as well: the authority_code / wkt2:2019 pass-through, discarded json.Marshal errors, and a couple of nlreturn gaps that will likely fail CI lint.
Once the two main issues above are fixed, I’m happy to take another pass and approve.
| } | ||
| } | ||
|
|
||
| if crs == "OGC:CRS84" || crs == "EPSG:4326" { |
There was a problem hiding this comment.
EPSG:4326 and OGC:CRS84 aren't the same CRS — they have opposite axis order (CRS84 is lon/lat, EPSG:4326 is lat/lon). Collapsing them here means a geometry("EPSG:4326") field — which is exactly what PyIceberg emits via ga.wkb().with_crs("EPSG:4326") — reads back as geometry("OGC:CRS84"), so the schema silently changes on the read side.
The write path doesn't share the collapse either: icebergCRSToGeoArrowMetadata emits EPSG:4326 verbatim, so GeometryType{crs:"EPSG:4326"} → Arrow → GeometryType{crs:"OGC:CRS84"} and the two aren't Equals.
I'd drop the EPSG:4326 case here and treat it as a distinct authority code. If we genuinely want them unified, it has to be symmetric — normalize on both the write and read sides so the round trip is stable — and it shouldn't live silently at the schema layer. Either way a round-trip regression test for EPSG:4326 specifically would lock it down. wdyt?
There was a problem hiding this comment.
EPSG:4326 and OGC:CRS84 aren't the same CRS
They are for the purposes of the GeoArrow and Parquet specifications, which explicitly define the axis order for these cases
There was a problem hiding this comment.
I believe that a round trip regression test is definitely in order for this behavior.
@paleolimbot, is the asymmetric behavior here expected or should we collapse "EPSG:4326" at the write level too? (i.e. add a conditional in icebergCRSToGeoArrowMetadata)
There was a problem hiding this comment.
The ideal behaviour is that all of these become the Iceberg default CRS:
"crs": "EPSG:4326"and"crs": "OGC:CRS84"(case insensitive)"crs": {..., "id":{"authority": "OGC", "code": "CRS84"}"crs": {..., "id":{"authority": "EPSG", "code": 4326}-
"crs": {..., "id":{"authority": "EPSG", "code": "4326"}
When reading an iceberg default CRS to GeoArrow, emit "crs": "OGC:CRS84". I think your PR is currently missing this case.
This asymmetry is helpful to improve the compatibility of iceberg tables (e.g., for readers that can't or don't want to understand CRSes and only handle the default case).
| } | ||
| } | ||
|
|
||
| if strings.HasPrefix(strings.ToLower(crs), "projjson:") { |
There was a problem hiding this comment.
This panics on projjson, but the symmetric read function geoArrowCRSToIcebergCRS returns a clean error for the same case. The asymmetry is the problem: VisitGeometry/VisitGeography have no error return and nothing recovers, so a GeometryType with a projjson: CRS detonates the whole conversion at TypeToArrowType rather than surfacing an error.
I'd return an error here and thread it through the same way the read path does, so a projjson CRS fails gracefully instead of crashing the process.
|
|
||
| iceType, err := geoArrowMetadataToIcebergType(wkb.Metadata()) | ||
| if err != nil { | ||
| panic(fmt.Errorf("%w: %v", iceberg.ErrInvalidSchema, err)) |
There was a problem hiding this comment.
The %v on err drops it out of the wrapped chain, so errors.Is/errors.As against the underlying error won't work downstream. I'd use %w for both:
panic(fmt.Errorf("%w: converting geoarrow metadata: %w", iceberg.ErrInvalidSchema, err))| return "srid:" + crs, nil | ||
| case geoarrow.CRSTypePROJJSON: | ||
| return "", errors.New("geoarrow CRS type projjson not supported yet") | ||
| default: |
There was a problem hiding this comment.
CRSTypeAuthorityCode and CRSTypeWKT22019 both fall into default and get returned as-is. For authority_code that's almost right but loses the type tag; for WKT2:2019 it hands a multi-KB WKT2 blob straight back as the Iceberg CRS string, which isn't a valid CRS identifier.
I'd add an explicit CRSTypeAuthorityCode case, and for CRSTypeWKT22019 either error like projjson does or reduce it to the authority code — whichever we pick, a test for each so the behavior is pinned.
There was a problem hiding this comment.
The CRSType, if it's coming from JSON, is just a hint, is not required, and is often absent. Here you probably want to:
- Check if crs is a JSON object. If it is, check the
idmember and paste togethercrs["id"]["authority"],:, andcrs["id"]["code"]. If any of those are missing, error for an unsupported CRS> - Check if crs is a string. If it's shorter than 32 characters, let it through verbatim. There's no official restriction on allowed characters in authorities or codes but the length check should reject anything questionable.
|
|
||
| raw, _ := json.Marshal(crs) | ||
|
|
||
| return geoarrow.Metadata{ |
There was a problem hiding this comment.
For an authority-code CRS like EPSG:4326 this emits {"crs":"EPSG:4326"} with no crs_type, but the GeoArrow spec wants crs_type: "authority_code" for AUTHORITY:CODE strings. Without it the encoding is ambiguous, and combined with the collapse in geoArrowCRSToIcebergCRS it's the other half of the EPSG:4326 round-trip break.
I'd set CRSType: geoarrow.CRSTypeAuthorityCode when the CRS matches the authority:code shape, so the write side is unambiguous and pairs cleanly with an explicit authority_code read case.
There was a problem hiding this comment.
"crs_type": "authority_code" is optional and no consumer actually requires this, but it is polite to include it (I put this in the Arrow C++ export path).
| return geoarrow.NewMetadata() // srid:0 maps to omitted GeoArrow CRS | ||
| } | ||
|
|
||
| raw, _ := json.Marshal(id) |
There was a problem hiding this comment.
errcheck is enabled outside _test.go, so the discarded errors here and at the json.Marshal(crs) below will fail CI. Marshaling a string can't actually fail, but the linter doesn't know that — I'd either build the raw JSON without the error return (e.g. json.RawMessage(strconv.AppendQuote(nil, id))) or add a //nolint with a one-line why.
|
|
||
| func geoArrowCRSToIcebergCRS(meta geoarrow.Metadata) (string, error) { | ||
| if len(meta.CRS) == 0 && meta.CRSType == "" { | ||
| return "srid:0", nil |
There was a problem hiding this comment.
A bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.
Separately, an empty meta.CRS with a non-empty CRSType skips this early return and falls into the switch, so CRSTypeSRID yields "srid:" with an empty id and GeometryTypeOf accepts it silently. I'd return the OGC:CRS84 default for the bare case and guard the empty-CRS-with-CRSType combination, with a test for bare geoarrow.wkb.
There was a problem hiding this comment.
bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.
This is the correct behaviour: the GeoArrow default does not equal the Iceberg default. PyIceberg is probably wrong here.
Separately, an empty meta.CRS with a non-empty CRSType skips this early return
This should be fixed...the CRSType can actually just be ignored for the purposes of this function (it's purely a hint)
| } | ||
| } | ||
|
|
||
| func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata { |
There was a problem hiding this comment.
Small thing while we're here: strings.ToLower(crs) runs twice (here and for the projjson check) and the slice crs[len("srid:"):] indexes the original-case string after a lowercased prefix check. I'd hoist lower := strings.ToLower(crs) once and reuse it. Also worth noting the EPSG:4326 match in the read function is case-sensitive, so epsg:4326 falls through — strings.EqualFold would close that.
paleolimbot
left a comment
There was a problem hiding this comment.
A few more details but this is looking good!
| } | ||
| } | ||
|
|
||
| if crs == "OGC:CRS84" || crs == "EPSG:4326" { |
There was a problem hiding this comment.
EPSG:4326 and OGC:CRS84 aren't the same CRS
They are for the purposes of the GeoArrow and Parquet specifications, which explicitly define the axis order for these cases
|
|
||
| func geoArrowCRSToIcebergCRS(meta geoarrow.Metadata) (string, error) { | ||
| if len(meta.CRS) == 0 && meta.CRSType == "" { | ||
| return "srid:0", nil |
There was a problem hiding this comment.
bare ga.wkb() field with no CRS — the PyIceberg default and what older clients emit — reads back as geometry("srid:0") rather than geometry(), which isn't Equals to a default-CRS geometry.
This is the correct behaviour: the GeoArrow default does not equal the Iceberg default. PyIceberg is probably wrong here.
Separately, an empty meta.CRS with a non-empty CRSType skips this early return
This should be fixed...the CRSType can actually just be ignored for the purposes of this function (it's purely a hint)
| return "srid:" + crs, nil | ||
| case geoarrow.CRSTypePROJJSON: | ||
| return "", errors.New("geoarrow CRS type projjson not supported yet") | ||
| default: |
There was a problem hiding this comment.
The CRSType, if it's coming from JSON, is just a hint, is not required, and is often absent. Here you probably want to:
- Check if crs is a JSON object. If it is, check the
idmember and paste togethercrs["id"]["authority"],:, andcrs["id"]["code"]. If any of those are missing, error for an unsupported CRS> - Check if crs is a string. If it's shorter than 32 characters, let it through verbatim. There's no official restriction on allowed characters in authorities or codes but the length check should reject anything questionable.
|
|
||
| raw, _ := json.Marshal(crs) | ||
|
|
||
| return geoarrow.Metadata{ |
There was a problem hiding this comment.
"crs_type": "authority_code" is optional and no consumer actually requires this, but it is polite to include it (I put this in the Arrow C++ export path).
03ba187 to
e4fbac6
Compare
|
Hi @laskoviymishka @paleolimbot, thank you guys for giving my PR a second look. I appreciate the comments! I believe I have addressed all comments but feel free to correct me if I missed anything!
all map to default Iceberg CRS.
|
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you for this! A few details but this is looking great from the GeoArrow metadata side (with apologies if I've misunderstood any of the Go here)
laskoviymishka
left a comment
There was a problem hiding this comment.
Nice work on this round, most of the earlier feedback is in now:
- EPSG:4326 ↔ OGC:CRS84 collapse is symmetric
authorityCodeCRSregex is anchoredCRSTypePROJJSONhard-error branch is gone- test split / naming looks cleaner
%wdouble-wrap,errcheck, planar-geography docs, andEdgePlanar→Geometryare cleaned up
I’d hold before merge, mostly for one real bug.
The SRID write path slices the identifier out of the lowercased CRS string, so non-numeric SRIDs get case-folded. srid:MyDB round-trips as srid:mydb. Numeric SRIDs are fine, which is probably why this stayed hidden. Slicing from the original crs should fix it. I’d also add a srid:4326 case in typeCases to cover the common path.
One other thing: go-geom is now a production dependency for test-only code. I’d lean toward moving WKTToWKB into a test package, but that’s not blocking.
The rest is cleanup: dead len(meta.CRS) guards, EqualFold on an already-lowercased string, unreachable codeNum.String() branch, checkCRSSJSON typo, and an invalid-WKT negative test.
One aside: the projjson write path still panics rather than returning an error. Fine if it only goes through the visitor recover path, but any direct caller would still crash.
Once those are in, happy to take another pass.
| func icebergCRSToGeoArrowMetadata(crs string) geoarrow.Metadata { | ||
| lowerCRS := strings.ToLower(crs) | ||
| if strings.HasPrefix(lowerCRS, "srid:") { | ||
| id := lowerCRS[len("srid:"):] |
There was a problem hiding this comment.
The id is sliced out of the lowercased string, so a non-numeric SRID gets silently case-folded: srid:MyDB goes in, mydb gets marshalled, and it round-trips back as srid:mydb. Numeric SRIDs are fine, which is why it's latent.
I'd slice the original crs instead — id := crs[len("srid:"):] — and keep the HasPrefix check on lowerCRS.
The iceberg_to_arrow_round_trip loop only iterates typeCases, which has geomSRID0 but no non-zero SRID, so the common path isn't exercised either. Adding a srid:4326 case to typeCases covers the happy path and would have caught this — worth pairing the fix with that test case.
There was a problem hiding this comment.
good catch, will update the tests to include this.
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| package internal |
There was a problem hiding this comment.
This is package internal, and WKTToWKB's only callers are _test.go files, but go-geom is now a direct require in go.mod — so every consumer of table/internal (scanner, DV writer) transitively pulls it into the production module graph.
I'd move this into a _test.go file in package internal_test (the test already lives there) or a small testutil package, so go-geom stays test-only. Non-blocking from my side, and I haven't seen it raised on the threads — if the maintainers are fine carrying it that's a reasonable call — but I'd lean toward keeping it out of the production deps.
| } | ||
|
|
||
| switch { | ||
| case checkCRSString(meta.CRS): |
There was a problem hiding this comment.
A small cleanup cluster in this function:
The if len(meta.CRS) > 0 wrappers inside both this checkCRSString arm and the checkCRSSJSON arm below are dead — the guard at the top of the function already returns on empty, and the check* funcs only return true for non-empty input (staticcheck SA9003). I'd drop both wrappers.
On the write path (line 1984), strings.EqualFold(lowerCRS, "EPSG:4326") runs a case-fold against an already-lowercased string — lowerCRS == "epsg:4326" is enough.
And in the JSON-object code parsing (line 1923), || codeNum.String() == "" is unreachable — a successful json.Number unmarshal is never empty. I'd drop just that clause; the trailing if code == "" is reachable for {"code":""}, so leave that one.
| // WKTToWKB is a helper which converts Well Known Text (WKT) to Well Known Bytes (WKB). | ||
| // Note that return bytes are little endian. | ||
| func WKTToWKB(s string) (geoarrow.WKBBytes, error) { | ||
| geometry, err := wkt.Unmarshal(s) |
There was a problem hiding this comment.
geo_codec_test.go is all happy-path — no invalid-WKT case. I'd add a wantErr row (e.g. "not wkt") so the error path is exercised. While we're here, the wkt.Unmarshal / wkb.Marshal errors return bare; wrapping them (fmt.Errorf("parse WKT: %w", err)) makes failures legible at the call site.
| return len(b) > 0 && b[0] == '"' | ||
| } | ||
|
|
||
| func checkCRSSJSON(rawCrs json.RawMessage) bool { |
There was a problem hiding this comment.
Minor: checkCRSSJSON has a double S — rename to checkCRSJSON at the def and the call site.
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
Signed-off-by: happydave1 <dzhao2004@gmail.com>
|
Thanks for the comments and review guys. I went through and should have addressed all comments for this round.
Thank you again for the catches and let me know if I missed anything. Thanks! |
paleolimbot
left a comment
There was a problem hiding this comment.
Just nits from my end. Thank you!
| // Always add an edge to differentiate between Geography and Geometry arrow fields. | ||
| // Note that the edge convention is a best-effort hint and planar geography from other clients won't round-trip through Arrow alone. |
There was a problem hiding this comment.
I think this comment is misleading (planar geography is not a thing, and this is not a hint, it affects correctness).
| // Always add an edge to differentiate between Geography and Geometry arrow fields. | |
| // Note that the edge convention is a best-effort hint and planar geography from other clients won't round-trip through Arrow alone. | |
| // Always add an edge to differentiate between Geography and Geometry arrow fields. |
Signed-off-by: happydave1 <dzhao2004@gmail.com>
laskoviymishka
left a comment
There was a problem hiding this comment.
I think it's in a good shape now, lets merge it! ![]()
Fixes #991.
Added metadata support in arrow_utils.go, created a
WKTToWKBhelper intable/internal/geo_codec.gowhich uses go-geom to convert WKT to WKB, and added a round trip Arrow to Parquet test which tests if geoarrow extension metadata survives. Also tests Iceberg schema to Arrow schema round trips.